Nuclei detection is an initial step in the development of computer aided diagnosis and automated tissue grading schemes in the context of digital pathology images. Accurate nuclei detection in images that have poor staining or noise is a challenging task. Nuclear clusters that result from tissue sectioning artifacts also increase the challenge in accurately detecting nuclei. Manually identifying cellular nuclei, including identifying the location and extent of melanocyte invasion or breast cancer nuclei, is subjective and time consuming.
Uneven absorption of staining dyes by tissue, variations in staining procedures across different labs, or variations in exposure time for stain absorption by tissue, may result in significant differences in the quality of biochemical tissue staining and adversely affect the appearance of tissue and associated histologic primitives. Nuclei may appear to be clumped together, or appear to be blended in with the stroma, resulting in under-emphasis of nuclear boundaries. Uneven or imprecise staining may also result in hollow cores, further confounding nuclei detection.
Conventional approaches to nuclei detection and segmentation make assumptions about the staining quality of tissue and resulting image quality, and therefore may yield sub-optimal detection performance if the tissue preparation and staining is less than ideal or fails to meet the assumptions. Gradient based detection approaches require model initialization and yield worse results when used with poorly stained images. Supervised deep learning approaches are hindered by the often heterogeneous appearance of nuclei in different cohorts and the frequent availability of only a small number of representative training samples. Color and texture based approaches require consistent color or texture appearance in individual nuclei in order to work optimally. Thus, since nuclei detection is often a critical step in computer aided diagnosis and prognosis schemes in the context of digital pathology images, it would be beneficial to more accurately detect and count nuclei in histopathological images.